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Abstract. The heart of every Monte Carlo simulation is a source of high quality 
random numbers and the generator has to be picked carefully. Since the "Ferrenberg 
affair" [3] it is known to a broad community that statistical tests alone do not suffice 
to determine the quality of a generator, but also application-based tests are needed. 
With the inclusion of an extensible random number library and the definition of 
a generic interface into the revised C++ standard [1] it will be important to have 
access to an extensive C++ random number test suite. Most currently available test 
suites are limited to a subset of tests are written in Fortran or C and cannot easily 
be used with the C++ random number generator library. 

In this paper we will present a generic random number test suite written in 
C++. The suite is based on the Boost reference implementation [2] of the forthcom- 
ing C++ standard random number generator library. The Boost implementation so 
far contains most modern random number generators. Employing generic program- 
ming techniques the test suite is flexible, easily extensible and can be used with 
any random number generator library, including those written in C and Fortran. 
Test results are produced in an XML format, which through the use of XSLT trans- 
formations allows extraction of summaries or detailed reports, and conversion to 
HTML, PDF, PostScript or any other format. 

At this time, the test suite contains a wide range of different tests, including 
the standard tests described by Knuth [5], Vattulainen's physical tests [14], parts 
of Marsaglia's Diehard [6] test suite, and a number of number of newer test such 
as Gonnet's repetition test [4] 



1 Introduction 

Testing random number generators is a really important part in the selec- 
tion process for an adequate random number generator. But testing random 
number generators is also a very serious problem and not so easy to manage. 
In this paper we will present a generic framework (random number gener- 
ator test suite, RNGTS) to manage, perform and analyze random number 
generator tests. The framework is written in C++ and makes an extensive 
use of the generic programming technique. This technique allows the usage 
of different data types and supports all different random number generators 
which fulfill the specification in the upcoming C++ standard for random num- 
ber generators. An other important point is the flexibility of the framework. 



Our attempt offers a flexible and simple extensible solution for further tests 
and other generators. Using XML to format the output opens a wide field of 
standardized transformations to other formats. 

1.1 Random Number Generator Tests 

Using a random number generator is one thing, the knowledge about its 
performance another. Testing the performance means first thinking about 
the expected properties of the random numbers, for example its application 
specific requirements. In a second step the most suitable tests have to be 
chosen. This selection is not easy and often people make "overall" tests of 
the generator. The problem is that we probably know what the test does, but 
one rarely knows which one the sensitive parameters in the application are. 
So, it is preferable to run more tests than fewer. 

The best known source is Knuth's book [5] , but there are many other tests 
available, some included in test benches, other as single procedures. The best 
known test bench is probably the Diehard test suite from Marsaglia [7] of 
which a new version has been released [8] . An other test bench, more special- 
ized on physical tests, is available from Vattulaincn [13]. Furthermore there is 
the SPRNG library [9] for parallel generators. In addition to the numerated 
test benches there are a lot of stand alone procedures like "Maurers Univer- 
sal Test" [10] or the "Repeating Time Test" [4]. Different tests are a dime a 
dozen, so we collected the most popular ones in table 3. All tests in the table 
having a "Class Name" are available in the RNGTS framework. 

1.2 Generic Programming 

Generic programming in C++ became popular with the introduction of the 
Standard Template Library (STL) into the C++ Standard. The STL is a 
collection of often used Containers and Algorithms. The principle of generic 
programming is simple, instead of fixing the view onto particular data struc- 
tures, the main interest are lying on the algorithms. This requires the intro- 
duction of a special data type handling mechanism. In C++ this technique is 
called template programming. 

One simple example is the maximum operator, the ordinary maximum 
function. In an ordinary way, the maximum operator may be defined as fol- 
lows. 

const int maxCconst intfic a, const int& b) 
{ return (a > b) ? a : b; } 

This method works well with integers, but we get into trouble if there are two 
doubles to compare. One solution is the replacement of all int's with double's. 
But the next problem occurs if the values are of type byte. The solution for 
this rapidly multiplying dilemma is, as mentioned above, templates. The 
specific data type is replaced by a generic placeholder which must be defined 
at compile time. Now, the maximum operator can be defined as followed. 



template <typename T> 

const T maxCconst T& a, const T& b) 

{ return (a > b) ? a : b; } 



The precondition for making this method work is that the comparison oper- 
ator ">" must be defined. 

The diversity of the STL gives an idea how powerful the usage of the 
template concept is. But this is only the tip of the iceberg, one can do much 
more. There is a technique called "Template Metaprogramming" , based on 
templates. This syntax allows compile time programming, e. g. nifty things 
like compile time if 's or decisions about used data types. A number of different 
applications of this template technique can be found in the BOOST library 
[2] as "Generic Programming" and "Template Metaprogramming" . 

1.3 The Boost library 

This section gives a short description of the BOOST project and presents the 
"BOOST Random Number Library". 

The following quotation from the Boost home page describes the project 
in a few words. 

The Boost web site provides free peer-reviewed portable C++ 
source libraries. The emphasis is on libraries which work well with 
the C++ Standard Library. The libraries are intended to be widely 
useful, and arc in regular use by thousands of programmers across a 
broad spectrum of applications. 

A further goal is to establish "existing practice" and provide refer- 
ence implementations so that Boost libraries are suitable for even- 
tual standardization. Ten Boost libraries will be included in the C++ 
Standards Committee's upcoming C++ Standard Library Technical 
Report as a step toward becoming part of a future C++ Standard. 

The project contains already a huge collection of libraries and one of them 
is the Random library. The Random library is one of the libraries included in the 
upcoming C++ Standard. Working with this random library means working 
with the future of C++. 

The Random library assembles a specific random number generator from 
to parts. One part is an "engine" (the raw random number generator), the 
other part is a "distribution" . This two parts together are represented by a so 
called variate_generator. For both parts there are a number of predefined 
types. Some examples are shown in table 1 and 2. 

To manufacture a desired generator, one has only to specify the desired 
components and put them together. The code snippet below shows an exam- 
ple. 



Boost Random Number Engine Example 



linear_congruential 
additive_combine 
laggecLf ibonacci 
mersenne .twister 
shuf f le_output 



GGL 

L'Ecuyer 1988 

R1279 

MT19937 

Bays-Durham shuffle 



Table 1. Some BOOST random library generators 



Boost Random Number Distribution Example 

unif orm_01 i£l,i£[0, 1) 

unif orm_int x G N 

unif orm_real x G R, x G [min, max) 

unif orm_on_sphere i£S° 

normal.distribution P(x) = _i e -(z-M) 2 /(2<r 2 ) 

exponential_distribution P{ x ) = Ae~ Aa: 



Table 2. Some BOOST random library distributions 



#include <boost /random . hpp> 

boost :: mtl9937 rng ; 

| boost : : unif orm_ int <> six ( 1 , 6) 



// the required libraries 

// the engine is a Mersenne Twister 
// the distribution maps to integer 1..6 



// glue the components together 

boost :: var iat e _gener at or <boost :: mt 19937 , boos t :: unif orm_ int <> > 
die (rng , six) ; 

int x = die () ; // use the generator. . . 



2 Random Number Generator Test Suite Framework 
2.1 Statistics 

After running a test one has to check if the obtained result is the expected 
one or something different. To classify the differences there are a variety of 
different statistical methods. There are three different statistical methods 
most frequently used. These are Chi-Square (x 2 ), Kolmogorov-Smirnov and 
Gaussian test. To give an introduction, there is a short description below. As 
the final output of each statistic, one or more probabilities in the [0, 1] range 
are expected. These values can provide a confidence level. 

Chi-Square (x 2 ) The x 2 -Test is perhaps the best known statistical test. It 
is based on a comparison between the empirical data and a theoretical 



distribution. The empirical data are the results of the random process. 



x 2 , 

Kolmogorov-Smirnov The \ 2 test is applied when observations can fall 
into a finite number of categories. But normally one will consider random 
quantities which may assume an infinite number of values. In this test, 
the random number generators distribution function F n (x) is compared 
to the theoretical cumulative distribution functionF(x). 

K+ = y/n max (F n (x) - F(x)) = max ( - - F{Xi) ) (2) 

— oo<a;<oo l<i<n \n J 

K~ = max (Fix) - F n (x)) = Vn max ( F(XA - (3) 

-oo<a;<oo l<i<n \ 71 J 

Gaussian The Gaussian test is a little different from the \ 2 or the Kolmogorov- 
Smirnov test. In these two tests the expected distribution function is com- 
pared with the measured distribution function and based on the differ- 
ence some indicators are calculated. In the Gaussian test a physical view 
is used. If a measurement is done, it is known that, even if the best tools 
are used, the result depends on a number of ruleless and uncontrolled 
parameters. These measurement errors are random and a combination 
of different single errors. This statistic calculates the probability for the 
deviation from the mean value. The used formula is 

P(x) = J-Jerf (4=z) (4) 



2 2 VV2 

where erf denotes the "error function" crf(z) = e~* dt. 

The RNGTS-Framework implements these three statistics as base classes 
for the tests. They are named as chisquare_test, ks_test and gaussian_test 
and implement the needed statistical methods. Using one of these classes is 
not mandatory, but recommended. A benefit is the possibility of using other 
helpers like iterate_test. 



2.2 Tests 

Well known sources for random number generator tests are the book of Knuth 
[5] and Marsaglias Diehard test suite [7] and [8]. But there are many more 
tests, perhaps not so nicely packaged as in the works mentioned above, but 
still useful. Each of these test checks a specific property of the tested gener- 
ator, e. g. the uniformity of binned values or the uniformity of 'continuous' 



values. The most popular tests are shown in table 3. The same table con- 
tains also the name of the class which implements the test, if it is already 
implemented. 

To make a test work with the RNGTS framework, one only has to im- 
plement a specific interface, or in other words, there are some methods in a 
test class which are needed by the framework. The implementation may be 
done from scratch or if one of the often used statistics is needed, by deriving 
from a given base class. 

Implementing a test: The following listing shows the base of each test, con- 
taining all required methods. This methods have to be implemented if the 
test is not derived from one of the provided test base classes. 

#include " xml_helper . h" // XML output functions 

class the_new_test 
{ 

public : 

the_new_test(. . .) ; 

template < class RNG > 
void run ( RNG& rng); 

std:: string test_name() const; 

template < class Input It er at or > 
void analyze ( xml _ st reaifc out, 

Input It er at or cl_begin, 

Inputlterator cl_end) const; 

void print .parameters C xml_stream& out) const; 

} 



constructor The constructor has to take all parameters needed for further 
calculations, e.g. the number of random numbers to test or the number 
of test runs. 

run The run method is the core of every test, this method has to run the 
test sequence and calculate the appropriate statistic. The generator to 
use is passed as rng and fulfills the Boost random number generator 
specification. 

test_name This method has to return the name of the test which is printed 
in the output. 

analyze The analyze method checks the test results against a given set of 
confidence levels, which are passed by iterators. The check also includes 
the decision if a test passed or failed at the confidence level. The output is 
written onto an XML stream. Several methods for producing valid XML 
output are provided. The output has to fulfill the XML Schema. 

print_parameters This method has to print all relevant parameters to the 
given XML stream. 



Test 



Class Name 



Equidistribution Test (Frequency Test) 

Gap Test 
Ising Model Test 
n- block test 
Serial Test 

Poker Test (Partition Test) 
Coupon collector's Test 
Permutation Test 
Run Test 

Maximum of t Test 

Collision Test (Hash Test) 

Serial correlation Test 

Birthday-Spacing's Test 

Overlapping Permutations Test 

Ranks of 31 x 31 and 32 x 32 matrices Test 

Ranks of 6 x 8 Matrices Test 

Monkey Tests on 20-bit Words 

Monkey Tests OPSO,OQSO,DNA 

Count the l's in a Stream of Bytes 

Count the l's in Specific Bytes 

Parking Lot Test 

Minimum Distance Test 

Random Spheres Test 

The Sqeeze Test 

Overlapping Sums Test 

The Craps Test 

Sum of distributions (for parallel streams) 
FFT 

Blocking Test 
2-d Random Walk 

Random Walkers on a line (Sjn Test) 

2D Intersection Test 

2D Height Correlation Test 

Repeating Time Test 

Gorilla Test 

CCD Test 

Maurers Universal Test 



ks_uniformity_test 

chisqr .uniformity .test 

gap.test 

ising_model_test 

n_block_test 

seriaLtest 

poker_test 

coupon_collector_test 

permutation_test 

runs_test 

max_of_t_test 

collision_test 

seriaLcorrelat ion_test 

birthday j3pacing_test 

overlapping_permutations_test 

bin_rank_chisqr_test 

bin_rank_ks_test 

monkey _20bit_test 

monkey_[OPSO | OQSO |DN A] .test 

count_ones_stream_test 

count_ones_bytes_test 

parking _lot_test 

minimum_distance_test 

random_sphere_test 

squeeze_test 

overlapping_sums_test 

craps_test 



random_walk_test 



height_corr2d_test 
repetition_test 
gorilla_test 
gcd_test 

maurers_universal_test 



Table 3. Available tests in the RNGTS framework. A detailed description or/and 
the original sources may be found in [12]. 



2.3 Other Utilities 



To perform a widespread analysis of a random number generator, the most 
important using is a variate of different tests. Occasionally, people want to 
test particular behavior of the generator, or they have to repeat a test mul- 
tiple times or for different seeding strategies, etc. A short overview of the 
possibilities for included utilities are itemized below. 

count_fails_test Repeats a test several times and count the number of fail- 
ings per confidence level 

iterate_test Repeats a test several times and calculates a Kolmogorov- 
Smirnov statistic from the results 

parallel_rng_imitator Combines multiple random number generators to 
one generator 

rng_bit_extract Interpret a random number as bit field and combine a se- 
quence of successive bits to a new number 

rng_bit_test Interpret a random number as bit field and generate sub-fields 
by shifting a specific mask over the field 

rng_file Reads random numbers from a binary file 

rng_wrapper Wrap external generators (e.g. Fortran, C) to use in the frame- 
work 

2.4 Results 

To rate a tested random number generator, one needs an overview over all 
test results with different seeds at different confidence levels. Additional in- 
teresting information to ensure the reproducibility of performed tests should 
also be saved. 

This variety of different information are stored in an XML structure. An 
XML structure allows transformations of the raw results to different target 
representations like HTML or I^TjtX. These two transformations are imple- 
mented using xslt (XML Stylesheet Language Translation). The ability of 
newest Internet browsers to format and display XML files makes the gener- 
ation of a report really simple. If a report in L^IpX style is required, an xslt 
processor (like xsltproc) has to be used. 

2.5 An Example 

To give an impression of the usage of the RNGTS framework, a short example 
is presented. As we can see here, the usage is intuitive and straight forward. 

#include <boost /random . hpp> // import the boost random library 

#def ine PRIKT_STATUS // print status information at run time 

#include <rngt s / rng_t e s t _suit e . h> // import the rngts framework 

#include <rngt s / chi sqr.unif ormi ty _t e s t . h> // import all required tests 

int main ( ) 
{ 



rng_test_suite <> rngts; 



// create test-suite 



// specify confidence levels 

rngts . add_confidence_level CO. 05) ; 

rngts . add_confidence_levelC0.95) ; 



// add 5% confidence level 
// add 95% confidence level 



rngts . add_seed (331) ; 
rngts . add_seed (667790) ; 



// add seed 331 
// add seed 667790 



// register random number generators to test 

// using Mersenne-Twister and RanLux from boost 

rngts . registering <boost::mtl9937>("mt -19937 "); 

rngts . register_rng<boost : : ranlux64_3_01 >("rl-64") ; 

// create and initialize a chisquare test object 
chisqr_unif ormlty_test chidOOOOO, 256); 
// register the chisquare test 

rngts . register_test <chisqr_uniformity_test>(chi) ; 

std: : of stream file_out ("results .xml") ; // file to write output 

try 

{ 

rngts . run_test (file_out , true); // run all tests 

} catch (std : : exception^ e) 
{ 

std::cout << "failure : " << e.whatO; 

} 

f ile_out . close () ; // close output file 



If the example above is compiled and executed, the file results. xml is 
written. A part of this file is printed below. The extract shows the results of 
a x 2 test of the Mersenne Twister 19937 generator. The test was performed 
with a seed of 331 and no "warmup" runs were executed. These are the con- 
figuration data of the generator section. In the test section, the needed test 
parameters are contained in the PARAMETERS block, and the ANALYZE block 
contains the test results. The results are included in the test name tag or in 
generalized result tags. Information whether a test passed or failed is repre- 
sented by tags named PASSED or FAILED which also contain the appropriate 
confidence level. 

<?xml vers ion = 11 1 . " ?><?xml - stylesheet href = " xml2html . xsl " type= " text /xsl " ? > 
<RNG_TEST_SUITE_RESULT dat e = " 2004 -04 -26 " > 
<RNG name="mt -19937 " warmup =" " > 
< SEED seed="331"> 

<TEST name=" Chi -Square -Uniformity -Test "> 



<PARAMETERS> 

<PARAMETER name="Number of Numbers" value =" 100000 "/ > 
<PARAMETER name="Number of Classes" value =" 256 "/ > 

</PARAMETERS> 

<ANALYZE> 

<CHI_SQUARE chi2="242.33" probabi 1 i t y= " . 706 " dof="255"> 

<PASSED confidenceLevel="0.05"/> 

<PASSED confidenceLevel="0.95"/> 
</CHI_SQUARE> 
</ ANALYZE > 



} 



</TEST> 
</SEED> 



</RNG> 

</RNG_TEST_SUITE_RESULT> 



3 Summary 



The presented RNGTS framework is a powerful but manageable tool to test 
the performance of C++ standard compatible random number generators. It 
provides a variety of random number generator tests an some other helpfully 
utilities. The ease of transforming XML to other formats allows postprocess- 
ing results to different representation like HTML or L^TeX. 

But, one has to keep in mind, the final decision about using a generator 
or not can not be done by any program, because the usability strongly de- 
pends on the application. But, this framework should be a help to simply the 
decision. 

4 Sources 

The RNGTS framework is available at http : //www . comp-phys . org/rngts. 
At this point one can also find a detailed description of the framework, some 
examples and related material. 
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